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Preface 


On  the  staff  of  the  Potomac  Institute  for  Policy  Studies  (the  "Institute")  are  many  employees 
who  have  come  to  us  after  long  and  successful  careers,  often  in  the  public  sector.  It  has  been 
an  Institute  tradition  to  ask  these  employees  to  spend  some  time  recording  their  thoughts  on 
matters  of  particular  interest  to  them.  The  choice  of  subject  is  guided  by  the  employee's 
depth  of  experience  as  well  as  his  or  her  concern  about  associated  issues.  In  many  cases,  the 
subject  is  also  of  great  interest  to  a  sponsor,  who  agrees  to  fund  the  resulting  monograph. 

In  response  to  an  unsolicited  proposal  submitted  in  early  2001,  the  0  ffice  of  Naval  Research 
(ONR)  awarded  a  grant  to  the  Institute  that  funded  research  for  this  monograph  on  the  need 
for  invigorating  the  Department  of  Defense's  (DOD)  science  and  technology  (S&T) 
management  and  investment.  D  rawing  from  many  years  of  experience  the  author,  Mr.  Hugh 
Montgomery,  provides  the  results  of  extensive  reading,  with  an  emphasis  on  numerous 
D  efense  Science  Board  studies  on  the  subject,  and  traces  the  history  of  the  D  epartment  of  the 
Navy  (DON)  Warfare  Center  System  to  illustrate  his  points.  He  concludes  with  a  plea  to 
develop  a  better  understanding  of  the  strengths  and  weaknesses  of  D  0  D 's  S&T  system  and  to 
begin  a  series  of  examinations  that  will  bring  beneficial  changes  to  a  crucial  aspect  of  our 
defense  strategy  -  developing  and  exploiting  the  best  technology  for  the  defense  of  our 
country.  Although  the  opinions  contained  are  the  authors  and  do  not  necessarily  represent 
the  views  of  the  sponsor,  we  hope  this  thoughtful  treatment  of  the  subject  will  contribute  to 
the  understanding  of  this  important  issue  area. 

Mr.  Montgomery  served  the  U.S.  Navy  for  over  thirty  years,  principally  in  conducting  and 
managing  research  and  development  (R&D ).  His  last  assignment  was  D  eputy  D  irector,  T est 
and  Evaluation  and  Technology  Reguirements,  Office  of  the  Chief  of  Naval  Operations 
(N091B).  With  impressive  expertise  in  science  and  technology,  public  and  private  sector 
research,  development  and  acguisition,  and  in  building  programs  and  management  systems, 
Mr.  Montgomery  was  instrumental  in  creating  the  Advanced  Technology  Demonstration 
(ATD )  and  Future  Naval  Capabilities  (FNC)  Processes.  In  his  last  assignment  he  served  as 
the  senior  D  epartment  of  Navy  (D  0  N )  career  civilian  for  reguirements,  assessment,  planning, 
budgeting  and  program  justification  to  Congress.  He  provided  oversight  of  a  $1.7  billion 
annual  S&T  and  supporting  R&D  investment. 

During  his  career,  Mr.  Montgomery  developed  an  in-depth  understanding  of  the  industry  R&D 
base.  He  served  as  the  first  Navy  Independent  Research  and  Development  (IR&D)  Program 
Manager,  with  oversight  and  assessment  responsibilities  for  $5B  annual  private  industry  R&D 
investment.  He  built  DON  interface  with  major  corporations  in  the  private  sector.  He  also 
developed  the  first  Planning,  Programming  and  Budgeting  System  (PPBS)  process  for  DON 
technology  programs  and  he  became  the  first  DON  principal  official  for  S&T  PPBS/ budget 
development. 

His  many  awards  include  the  Navy  Distinguished  Civilian  Service  Award  (highest  DON 
award),  which  he  received  in  1992  and  2000—  the  first  time  it's  been  presented  twice  to  the 


same  person.  He  was  also  awarded  the  Presidential  Meritorious  Executive  Award,  the  N avy 
Superior  Civilian  Service  Award  (Second  Highest  DON  Award),  and  the  Secretary  of 
D  efense's  Commendation/  Superior  Performance  Award.  He  holds  six  patent  awards. 


James  J.  Richardson,  Ph.D . 

Vice  President  for  Research 
Potomac  Institute  for  Policy  Studies 


Department  of  Defense 
Science  And  Technology  Invigoration 

Abstract 

Numerous  expert  studies  suggest  that  the  Department  of  Defense  (DOD)  and  the  national 
science  and  technology  (S&T)  base  have  been  in  a  slow  decline  since  the  Viet  Nam  War.  Reasons 
given  include  decreased  funding,  disestablishment  of  executive  level  sponsors,  and  growing 
competition  for  top-level  personnel.  This  report  examines  the  evidence  of  decline  and  offers  a  set 
of  metrics  that  may  be  used  to  conduct  an  objective  assessment  of  DOD  S&T  capabilities  and  of 
the  best  use  of  those  capabilities.  In  addition,  the  potential  for  invigorating  the  D  0  D  Research, 
Development,  Testing  and  Evaluation  (RDT&E)  base,  and  the  S&T  base  in  particular,  is  discussed 
in  the  context  of  the  same  metrics.  The  insights  and  thought  processes  necessary  to  develop, 
define,  and  most  importantly,  come  to  agreement  on  metrics  and  options  will  be  invaluable  in 
directly  pursuing  and  addressing  the  immense  problems  associated  with  optimizing  the  DOD's 
S&T  operations  for  today's  and  tomorrow's  world,  an  undertaking  beyond  the  scope  of  this  study. 

A  history  of  the  national  S&T  base  shows  profound  changes  over  the  past  30  years  or  more. 
Many  critics  of  today's  S&T  system  believe  these  changes  result  from  a  series  of  arbitrary  reactions 
to  trends  in  the  public  and  private  sectors.  Whether  this  is  true  or  whether  these  changes  simply 
reflect  inevitable  and  proper  adjustments  to  different  times,  it  is  clear  that  the  S&T  system  needs 
attention.  The  first  step  is  to  develop  a  means  to  analyze  the  complex  and  multifaceted  issues 
associated  with  the  national  enterprise  supporting  science  and  technology. 

This  paper  proposes  an  approach  to  such  an  analysis,  one  that  incorporates  metrics  tailored  to  the 
issues  at  hand.  Some  of  those  issues  are  examined,  using  the  history  of  the  Naval  laboratory  and 
warfare  center  system  as  a  case  in  point.  Then,  a  structured  approach  is  built  for  use  in 
understanding  the  pressures  on  the  S&T  system  and  how  that  system  should  react.  Most  notably, 
the  analysis  must  identify  those  factors  that  have  had  the  greatest  difect  on  the  technology  base 
and  which  of  these  factors  can  be  affected  by  government.  Once  this  is  done,  subseguent  studies 
can  perform  the  necessary  analyses,  formulate  solutions,  and  finally,  work  out  an  implementation 
strategy. 

Given  that  invigoration  of  the  DOD  S&T  base  is  necessary  at  some  level,  a  great  difficulty  is 
identification  of  which  aspects  of  the  system  reguire  invigoration  and  the  direction  that  such  an 
invigoration  should  take.  Unknown,  for  example,  is  the  appropriate  mix  of  technical  skills  to  meet 
the  challenges  of  the  new  century.  What  must  be  developed  ultimately  is  a  strategy  that  considers 
the  national  S&T  enterprise  and  the  contributions  of  both  public  and  private  sector  activities. 
This  paper  recommends  a  baseline  assessment  from  which  development  of  such  a  holistic  strategy 
can  begin. 

In  preparation  for  this  task,  a  survey  of  past  studies  of  the  DOD  and  Naval  S&T  system  has  been 
performed,  offering  a  summary  of  common  conclusions  and  recommendations. 
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Executive  Summary 


In  the  early  days  of  the  Reagan  defense  buildup,  Secretaiy  of  Defense  Caspar  Weinberger 
wrote, 

We  face  the  danger  of  losing  our  edge  because  we  have  not  adeguately  replenished  the 
reservoir  of  scientific  concepts  and  knowledge  to  nourish  future  technologies  during 
subseguent  years  of  fiscal  neglect  of  defense  research  and  development.  G  iven  these 
circumstances,  we  must  systematically  replenish  that  scientific  reservoir,  using  the 
unigue  and  diverse  strengths  of  the  United  States  scientific  community  ...  Given  the 
relatively  long  lead  time  between  fundamental  discovery  and  applying  such  knowledge 
to  defense  systems,  the  true  measure  of  our  success  ...  may  not  be  apparent  for  several 
decades.  When  the  'moment  of  truth'  arrives,  we  cannot  afford  to  be  found  wanting. 

Thus  we  must  revitalize  the  productive  partnership  between  the  university 
community,  industry,  and  the  DOD  in-house  laboratories.1 

President  John  Kennedy's  announcement  in  1962  of  the  national  goal  to  land  an  American  on 
the  moon  and  return  him  safely  to  earth  before  the  end  of  the  decade  launched  a  science  and 
technology  (S&T)  golden  age  for  the  United  States.  For  science  and  engineering  graduates, 
there  was  the  promise  of  an  exciting  time  of  discovery  and  an  opportunity  to  serve  their 
nation  at  the  same  time.  The  level  of  S&T  investment  was  at  an  all-time  high.  Specific 
incentives  for  defense-related  employment  included  the  promise  of  occupational  draft 
deferments  for  scientists  in  defense  laboratories,  accompanied  by  a  sharp  downturn  in  non- 
Department  of  Defense  (DOD)  hiring  of  scientists  and  engineers  as  NASA  neared  its  moon 
goal.  In  the  late  1960s,  this  enabled  the  D  0  D  to  recruit  and  retain  an  exceptionally  talented 
group  of  young  professionals.  However,  after  the  Viet  Nam  War,  the  situation  changed 
radically.  DOD  laboratories  began  a  period  of  two  decades  of  minimal  new  hiring,  with  low 
attrition.  Although  total  DOD  S&T  funding  remained  relatively  constant,  Service  S&T 
spending  began  a  decline  that  continued  to  the  end  of  the  century;  their  S&T  focus  decreased 
as  D  efense  Agency  S&T  budgets  increased. 

The  law  supporting  industry  Independent  Research  and  Development  (IR&D)  changed  in 
1990,  and  IR&D  investment  shrank  to  less  than  half  of  its  previous  level.  The  Navy's 
Independent  Exploratory  D  evelopment  (IED )  Program,  designed  to  support  the  best  ideas  of 
the  best  scientists  and  engineers,  was  cancelled.  Near  the  century's  end,  the  rate  of  hiring  of 
DOD  scientists  and  engineers  increased,  but  with  intense  private  sector  competition  for  the 
best  and  brightest  added  to  the  historically  cumbersome  and  inflexible  regulations  for  Civil 
Service  hiring  and  noncompetitive  pay  rates.  Added  also  was  a  growing  public  disinterest  in 
pursuing  S&T  curricula  and  an  attitude  prevalent  among  many  leaders  that  in-house  DOD 
S&T  investment  was  not  reguired,  i.e.,  that  the  private  sector  could  and  would  provide  the 
technology  as  necessary. 

At  the  beginning  of  the  twenty-first  century,  an  entire  generation  of  defense  science  and 
engineering  talent  is  preparing  for  retirement.  An  important  issue  for  immediate  study  is  the 
impact  of  the  departure  of  this  generation  without  capable,  trained  relief  to  maintain  the 
technical  corporate  memory,  i.e.  the  potential  for  loss  or  significant  degradation  within  the 
next  decade  of  U.S.  technological  leadership  in  areas  of  critical  national  interests.  It  can  be 
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argued  reasonably  that  new  blood  in  the  system  may  introduce  an  era  of  fresh  thinking,  but  it 
is  difficult  to  escape  the  conclusion  that  the  potential  exists  for  serious  loss  of  capability, 
particularly  in  the  "art  of  science"— things  that  are  learned  from  experience  and  not  from 
formal  education.  Should  this  loss  occur  without  a  strategy  to  compensate,  the  D  0  D  S&T 
engines  could  stumble,  and  perhaps  grind  to  a  halt.  Indeed,  there  is  evidence  that  this  is 
happening. 

There  is  no  shortage  of  studies  of  the  D  0  D  Research,  D  evelopment,  T esting,  and  Evaluation 
(RDT&E)  infrastructure  over  the  past  two  decades.  In  fact,  it  is  difficult  to  filter  and  distill 
the  information  overload  into  what  are  highly  consistent  conclusions.  This  report  highlights 
studies  of  greater  prominence  or  depth,  focusing  in  particular  on  the  reports  of  the  Defense 
Science  Board  (DSB)  of  the  past  decade.  Although  only  a  small  sample  of  the  available 
information  is  highlighted,  the  degree  of  consistency  among  the  studies  is  exceptionally  high, 
including  even  those  studies  conducted  prior  to  the  Reagan  defense  buildup  of  the  1980s. 
Many  began  with  a  basic  assumption  that  D  0  D  laboratories  are  essential,  and  did  not  attempt 
to  investigate  or  justify  their  existence  within  the  D  0  D  and  national  system. 

0  f  the  studies  that  addressed  the  need  for  an  independent  in-house  RD  T &E  capability,  points 
of  agreement  on  their  value  to  D  0  D  include: 

•  To  help  D  0  D  and  the  Services  to  be  smart  buyers; 

•  To  ensure  the  ability  to  respond  rapidly  in  times  of  crisis; 

•  To  support  facilities  and  capabilities  not  practical  in  the  private  sector; 

•  To  execute  S&T  programs  and  maintain  the  long  term  view; 

•  To  provide  direct  technical  support  to  the  warfighter;  and 

•  To  maintain  technical  co rpo rate  memo ry. 

V irtually  all  of  the  studies  recognized  that  the  laboratory  hiring  reductions  of  the  1970s- 1 980s, 
after  several  years  of  significant  "staffing  up,"  created  a  predictable  future  loss  of  technical 
competence  and  technological  leadership.  The  dominant  points  of  concern  include: 

•  The  DOD  laboratory  system  has  declined  in  numbers  by  approximately  35  percent 
since  1990. 

•  The  DOD  laboratory  system  is  likely  to  lose  by  retirement  within  five  years  a 
generation  of  experienced  and  talented  staff,  with  no  trained  backup  to  carry  forward. 

•  Civil  Service  hiring  practices  and  policies  severely  limit  recruitment  of  the  best 
personnel,  either  in  career  or  appointee  status. 

•  The  current  S&T  funding  level  inhibits  funding  incentives  for  recruitment  of  the  "best 
and  brightest,"  increases  risk  for  the  future  and  raises  acguisition  costs. 

The  dominant  recommendations  include: 

•  DOD  must  augment  its  senior  staff  through  contract  staff  and  the  Intergovernmental 
Personnel  Act  (IPA)  Mobility  Program  from  industry,  academia  and  non-profits. 

•  DOD  must  utilize  and  exploit  commercial  technology  wherever  and  whenever 
possible. 

•  DOD  must  act  guickly. 
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This  report  addresses  the  specific  case  of  the  need  for  invigo ration  of  the  S&T  base  within  the 
major  components  of  the  DOD  S&T  enterprise.  A  primary  objective  is  to  investigate 
methodologies  to  ensure  the  "highest  and  best"  use  of  the  technical  capabilities  of  the  D  0  D 
laboratories  in  today's  changing  environment,  including  optimizing  the  necessary  public- 
private  sector  relationships.  The  second  objective  is  to  ensure  that  the  technical  capabilities  of 
the  enterprise  remain  sufficiently  robust  to  meet  the  needs  of  the  DOD  today  and  in  the 
future.  A  longer-term  objective  is  to  determine  what  is  reguired  of  the  enterprise  to  satisfy  the 
needs  of  the  D  0  D -after- next.  Finally,  the  most  important  challenge  is  to  determine  how  best 
to  attract  the  talent  to  ensure  technical  excellence  in  the  future.  (Note:  These  objectives  are 
likely  to  apply  egually  to  government  agencies,  defense  contractors,  and  other  elements  of  the 
private  sector.  If  so,  only  a  top-down,  holistic  strategy  will  succeed.) 

The  recent  economic  downturn,  hiring  cutbacks  in  the  commercial  technology  sector  and 
especially  the  events  of  September  11, 2001,  have  provided  a  D  0  D  employment  environment 
similar  to  that  of  thirty-five  years  ago.  A  number  of  other  independent  factors  have  aligned  to 
create  the  right  timing  and  to  provide  an  opportunity  for  D  0  D  S&T  invigoration.  First,  and 
most  important  of  these,  is  the  time  urgency  for  loss  of  irreplaceable  senior  staff.  Well- 
documented  from  multiple  studies  is  the  fact  that  the  highly  talented  generation  hired  in  the 
Viet  Nam  era  will  retire  from  the  system  within  five  years,  in  many  cases  without  trained  relief 
to  preserve  the  corporate  memory.  At  least  three  D  SB  Task  Force  reports  published  in  the 
year  2000  express  concern  for  the  loss  of  capability  within  the  DOD  laboratory  system  and 
propose  significant  remedial  actions.2,3,4  These  reports  echo  concerns  expressed  over  the  past 
two  decades.  Second,  a  new  forcing  function  to  address  the  issue  is  contained  in  Section  913 
of  the  Conference  Report  accompanying  the  National  Defense  Authorization  Act  (NDAA) 
for  Fiscal  Y  ear  2000  (Appendix  A).  This  legislation  provides  for  the  D  O  D  an  opportunity  (1) 
to  determine  the  state  of  S&T  capability  within  the  S&T  enterprise  and  (2)  to  develop  a  long- 
range  plan  (including  both  public  and  private  sectors)  to  ensure  the  maintenance  of  essential 
capabilities.  A  third  factor  is  the  change  in  national  administration,  which  invariably  leads  to 
evaluation  of  the  infrastructure  and  changes  to  policies  of  the  past  administration.  Finally, 
there  is  growing  recognition  in  the  military  and  across  the  public  and  private  sectors  of  the 
vital  importance  of  modernization,  scientific  discovery  and  innovation  to  the  U.S.  national 
defense  and  economic  interests. 

This  study  developed  metrics  for  near-term  assessment  of  the  state  of  S&T  in  the  DOD 
laboratory  system  and  related  Federally  Funded  Research  and  Development  Centers 
(FFRD  Cs).  Past  studies  of  assessment  metrics  agree  that  objective  evaluation  of  Research  and 
D  evelopment  is  very  difficult  at  best,  that  one  set  of  metrics  does  not  apply  universally,  and 
that  the  number  of  metrics  for  an  evaluation  should  be  small,  preferably  no  more  than  a  half- 
dozen.  The  following  assessment  metrics  are  proposed: 

1 .  Q  uality  and  military  impact  of  the  products  of  the  organization; 

2 .  O  rganizatio  nal  co  re  co  mpetencies; 

3.  V alue-added  of  the  organization; 

4.  Q  uality  of  the  technical  staff; 

5.  State  of  physical  resources;  and 

6.  State  of  fiscal  resources. 
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The  following  actions  are  recommended: 

1.  Conduct  an  assessment  of  the  DOD  S&T  enterprise,  and  the  DOD  laboratory  and 
warfare  center  system  in  particular.  The  assessment  should  be  consistent  with  the 
requirements  of  the  Section  913  legislation  and  with  requirements  input  of  the  Service 
Chiefs,  Systems  Commands,  and  Commanders- in-Chiefs  (CINCs).  Recommended 
metrics  for  assessment  are  the  six  listed  above. 

2.  Develop  a  strategy  to  ensure  maintenance  of  the  DOD  S&T  corporate  memory  as  the 
current  generation  of  senior  scientists  and  engineers  retires.  Examine  the  potential  new 
window  of  opportunity  to  recruit  the  "best  and  brightest"  into  the  DOD  laboratory 
system  and  defense- related  industry. 

3.  Develop  a  strategic  plan  for  the  DOD  S&T  infrastructure  of  the  future,  defining  the 
complementary  roles  of  the  Laboratory/  center  system,  academia,  FFRD  Cs  and  University 
Affiliated  Research  Centers  (UARCs),  and  private  industry.  The  system  of  the  future 
should  be  flexible,  innovative,  willing  to  promote  bypassing  the  next  generation  of 
technology,  and  able  to  support  the  military  forces  in  effective  utilization  of  the  systems 
and  technologies  of  U.S.  and  international  military  and  commercial  activities. 

4.  Establish  and  monitor  closely  a  6.2  base  program  within  the  D 0 D  laboratory  system  to 
encourage  discovery  and  innovation. 

5.  The  D  irector  D  efense  Research  and  E  ngineering  (D  D  R&E )  should  take  the  lead  to  carry 
out  the  above  recommendations. 


12 


I.  Introduction 


In  1973,  a  joint  review  of  Independent  Exploratory  Development  (IED)  was  conducted 
between  the  Naval  Research  Laboratory  (NRL)  and  the  Naval  Weapons  Laboratory  (now 
N aval  Surface  W arf are  Center  D  ahlgren  D  ivision).  IED  funding,  representing  the  Laboratory 
D  irectors'  only  internally  managed  Applied  Research  (6.2)  dollars,  disappeared  in  S&T  funding 
reductions  of  the  last  decade.  IED  dollars  were  few  and  very  precious,  allocated  only  to  the 
best  people  and  the  best  ideas  (Note:  There  has  always  been  and  there  will  always  be  more 
worthy  ideas  than  funds.).  A  project  considered  for  funding  during  that  review  was  a  radically 
new  concept  called  G  lobal  Positioning  System  (G  PS),  for  which  the  briefer  reguested  one  or 
two  work-years  of  funding  to  examine  feasibility.  G  PS  in  1973  represented  exceptionally  high 
technical  and  budget  risk,  but  the  panel,  with  absolutely  no  idea  of  any  unclassified  or  non¬ 
military  applications,  decided  the  technology  might  be  worthy  of  a  few  "sorcery  and  alchemy" 
dollars.  Almost  thirty  years  later,  G  PS  is  a  household  word  in  commercial  and  military 
applications,  and  the  impact  of  this  breakthrough  technology  finally  may  be  understood  and 
appreciated. 

In  the  new  millennium,  Naval  IED  funding  no  longer  exists,  the  Navy's  6.2  Applied  Research 
account  is  half  the  size  of  1973  and  the  focus  of  Naval  and  other  Service  warfare  centers  has 
become  shorter-term,  farther  from  science  and  technology  (S&T).  In  this  climate  it  is 
reasonable  to  guestion  (1)  whether  today's  breakthrough  ideas  with  levels  of  risk  comparable 
to  G  PS  inl973  would  be  seriously  considered  for  S&T  funding,  and  (2)  whether  the  current 
capability  of  D  0  D 's  in-house  research  and  development  (R&D )  infrastructure  is  sufficient  to 
create  and  to  develop  such  "out-of-the-box"  concepts.  These  are  vital  guestions  from  both  a 
D  0  D  and  national  perspective. 

President  John  Kennedy's  announcement  in  1962  of  the  national  goal  to  land  an  American  on 
the  moon  and  return  him  safely  to  earth  before  the  end  of  the  decade  launched  what  was  to 
become  a  S&T  golden  age  for  the  United  States  government.  The  President's  timing  was 
outstanding.  Although  he  did  not  live  to  see  the  maturation  of  his  lofty  vision,  the  cause 
captured  not  only  the  imagination  of  the  American  people  as  a  whole,  but  also  the  imagination 
of  a  young  and  talented  baby-boom  generation  in  particular.  This  was  a  generation  bom  after 
World  War  II  (WWII)  and  raised  in  the  Eisenhower  years,  with  vague  recollections  of  Korea 
but  vivid  memories  of  Sputnik,  Berlin,  and  the  Cuban  Missile  Crisis.  For  this  generation,  a 
career  in  science  offered  the  ability  to  participate  in  a  great  and  exciting  adventure  and  serve 
the  nation  at  the  same  time.  This  was  made  possible  by  a  sufficient  level  of  national  S&T 
investment  to  support  this  adventure.  In  the  specific  case  of  the  Department  of  Navy 
(DON),  its  growing  1962  Applied  Research  (6.2)  budget  was  three  times  larger  than  the  draft 
FY  2002  President's  Budget  reguest  for  the  same  account. 

Far  less  noble  but  of  significance  in  the  latter  half  of  the  decade  was  the  fact  that  male 
scientists  graduating  with  B.S.  degree  or  advanced  degrees,  as  the  oldest  in  the  draft  pool,  were 
highly  vulnerable  to  be  drafted  and  sent  to  combat  in  the  nation's  most  unpopular  war.  The 
attractiveness  of  occupational  deferments  for  scientists  in  defense  laboratories  and  a  downturn 
in  non-DO  D  hiring  of  scientists  and  engineers  (S&Es)  as  the  National  Aeronautics  and  Space 
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Administration  (NASA)  neared  its  moon  goal  combined  with  grand  causes  and  patriotism  and 
created  a  1965-1971  buyer's  market  for  the  DOD,  an  opportunity  to  select  the  best  of  an 
exceptionally  talented  pool  of  young  professionals. 

By  the  end  of  the  Viet  Nam  W ar,  the  climate  had  changed  radically.  Federal  employment,  and 
defense  employment  in  particular,  was  not  viewed  with  respect,  and  morale  of  workers  sagged. 
Most  occupational  deferments  ended  with  the  draft  lottery  system,  and  employees  who 
worked  for  DOD  only  to  avoid  the  draft  (a  small  number)  moved  on.  DOD  laboratories 
began  a  period  of  two  decades  of  minimal  new  hiring,  low  to  moderate  attrition  and  a  talented 
work  force  entering  middle  age.  Service  S&T  spending  began  a  slow,  steady  decline  that 
continued  through  the  end  of  the  century.  Although  DOD  S&T  investment  remained 
relatively  constant  through  the  1980s  and  1990s,  creation  of  the  Strategic  Defense  Initiative 
initiated  a  significant  funding  migration  from  the  Services  to  Defense  Agencies  (Figure  1.). 


Fiscal  Year 

Figure  l.  Comparison  of  DOD  S&T  Accounts 


Note:  The  Services  and  Agencies  execute  significantly  different  S&T  investment  strategies. 
The  Services  tend  to  invest  their  share  (half)  of  S&T  -  the  high-risk  portion  of  the  DOD 
investment  portfolio  -  in  numerous  smaller  projects.  The  Agencies  tend  to  concentrate  S&T 
investment  into  a  smaller  number  of  projects  at  higher  funding  levels. 

In  the  early  Clinton  years  of  federal  downsizing,  introduction  of  new  scientists  and  engineers 
into  the  D  0  D  S&T  community  slowed  to  a  crawl  as  work  force  reductions  and  privatization 
goals  were  achieved  primarily  by  means  of  hiring  freezes,  attrition,  and  outsourcing.  In  the 
latter  years  of  the  Clinton  Administration,  some  hiring  of  scientists  and  engineers  resumed, 
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but  with  new  and  intense  private  sector  competition  for  the  best  and  brightest  added  to  the 
historically  cumbersome  and  inflexible  regulations  for  federal  hiring  and  noncompetitive  pay 
rates. 

Figure  2,  using  data  developed  in  1995  by  the  Navy,  graphically  illustrates  the  existence  of 
manpower  generation  gaps.  Similar  trends  are  known  to  exist  in  the  other  Service  RDT&E 
activities,  but  there  is  not  a  comparable  historical  database  for  the  enterprise  as  a  whole.  Thus 
potentially  serious  issues  of  manpower  and  capability  losses  are  not  readily  apparent. 


At  the  beginning  of  the  twenty-first  century,  an  entire  generation  of  defense  science  and 
engineering  talent  is  preparing  for  retirement.  An  important  issue  for  immediate  study  is  the 
impact  of  the  departure  of  this  generation  without  capable,  trained  relief  to  maintain  the 
technical  corporate  memory,  i.e.  the  potential  for  loss  or  significant  degradation  within  the 
next  decade  of  U.S.  technological  leadership  in  areas  of  critical  national  interests.  It  can  be 
argued  reasonably  that  new  blood  in  the  system  may  introduce  an  era  of  fresh  thinking,  but  it 
is  difficult  to  escape  the  conclusion  that  the  potential  exists  for  serious  loss  of  capability, 
particularly  in  the  "art  of  science"  -  things  that  are  learned  from  experience  and  not  from 
formal  education.  Should  this  loss  occur  without  a  strategy  to  compensate,  the  DOD  S&T 
engines  could  stumble,  and  perhaps  grind  to  a  halt.  Indeed,  there  is  evidence  that  this  is 
happening. 

This  study  proposes  an  examination  of  the  need  for  invigo ration  of  the  S&T  base  within  the 
components  of  the  DOD  S&T  enterprise  and  the  direction  that  invigo  ration  should  take.  A 
primary  objective  is  to  investigate  methodologies  that  ensure  the  "highest  and  best"  use  of  the 
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technical  capabilities  of  the  Service  laboratories  and  warfare  centers,  and  how  to  optimize 
them  for  today  and  tomorrow's  world.  This  implies  in  no  way  the  intent  to  maintain  status 
quo,  but  rather  the  intent  to  establish  the  most  appropriate  and  complementary  relationships 
among  the  public  and  private  sector  components  of  the  enterprise.  The  second  objective  is  to 
ensure  that  the  technical  capabilities  of  the  enterprise  remain  sufficiently  robust  to  meet  the 
needs  of  the  DOD  today  and  in  the  future.  A  third,  longer-term  challenge  is  to  determine 
what  is  required  of  the  enterprise  to  satisfy  the  needs  of  the  D  0  D -after-next,  i.e.,  what  must 
the  system  become  and  what  has  to  be  done  to  get  there?  Assuming  that  reliance  on  the 
private  sector  is  at  least  a  part  of  the  solution,  the  potential  for  private  sector  substitution  for  a 
portion  of  D  0  D 's  S&T  force  must  be  examined  closely  and  a  methodology  to  accomplish  the 
required  leveraging  must  be  developed.  This  leads  to  hard  questions  such  as  the  DOD 
personnel,  infrastructure,  and  skills  required  as  well  as  how  to  address  military-specific 
technologies  and  products  that  may  have  no  private  sector  interest. 

The  final  and  most  important  challenge:  how  to  ensure  technical  excellence  in  the  D  0  D  and 
nation- after- next  outside  the  climate  of  noble  causes,  excitement,  and  adventure  that  existed  in 
the  late  1960s  and  which  naturally  attracted  the  best  and  brightest  into  federal  service.  The 
recent  economic  downturn,  hiring  cutbacks  in  commercial  technology  employment  and 
especially  the  events  of  September  11,  2001  may  provide  a  forcing  function  similar  to  the 
scientific  employment  environment  of  thirty-five  years  ago.  If  so,  there  may  be  a  recruitment 
window  of  opportunity  for  D  0  D  and  the  defense-related  private  sector. 

This  report  will  examine  how  such  questions  can  be  addressed.  It  will  review  the  history  of 
the  N aval  warfare  center  system  as  a  case  study,  highlight  findings  and  recommendations  from 
past  R&D  infrastructure  studies,  and  propose  an  approach  for  assessment  and  potential 
invigo ration  of  the  S&T  capabilities  of  the  DOD  S&T  enterprise. 
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II.  History  of  the  Department  of  Navy 
Warfare  Center  System. 


This  section  documents  the  history  of  the  Naval  warfare  centers  as  a  case  study  of  the  decline 
in  in-house  DOD  S&T  capability.  Although  the  specific  details  vary  somewhat  the  concerns 
identified  for  the  Naval  activities  apply  equally  to  the  other  Service  components  of  the  D  0  D 
S&T  enterprise.  Thus  it  is  logical  to  assume  that  many,  if  not  all  of  the  potential  solutions  for 
D  epartment  of  Navy  problems  apply  to  all  Services. 

The  evolution  of  the  shore  establishment  and  acquisition  processes  of  the  D  epartment  of  the 
Navy  (DON)  is  characterized  by  124  years  of  relative  stability,  followed  by  35  years  of 
multiple,  significant  changes.  The  Naval  Bureaus  were  established  in  1842  as  a  replacement 
for  the  Board  of  Naval  Commissioners,  which  provided  management  over  Naval  material 
affairs.  After  124  years  of  survival  through  multiple  wars  and  political  administrations,  the 
Bureaus  were  replaced  in  1966  by  six  Systems  Commands  (SY  SCO  MS),  which  reported  to  the 
four- star  Naval  Material  Command  (NAVMAT). 5 

Under  this  new  structure  the  SY  SCO  MS  were  charged  with  conceiving,  developing,  acquiring 
and  logistically  supporting  Naval  platforms  and  their  weapons.  This  was  accomplished 
through  program  offices,  a  field  structure  and  industry,  with  technical  support  from  Naval 
R&D  activities  (commanded  by  military  officers  but  staffed  predominantly  by  Civil  Service 
employees)  and  university  laboratories.  The  Director  of  Navy  Laboratories  (DNL)  was 
established  to  provide  representation,  oversight  and  (loose)  coordination  of  the  Navy 
Laboratory  and  Warfare  Center  System.  The  university  laboratories  were  placed 
administratively  under  the  SYSCOM  for  which  they  did  the  majority  of  work.  The  Naval 
R&D  activities,  however,  although  they  received  the  majority  cf  their  funding  from  the 
SY  SCO  Ms,  reported  administratively  to  NAVMAT  and  DNL,  who  was  double- hatted  also  to 
the  Assistant  Secretary  of  the  Navy  for  Research,  Engineering  and  Systems.6 

At  the  same  time  as  NAVMAT  was  organizing,  the  Director,  Defense  Research  and 
Engineering  (DDR&E,  Dr.  John  Foster)  asked  Dr.  Leonard  Sheingold,  vice  president  of 
Sylvania  Electronic  Systems,  to  chair  the  Defense  Science  Board  (DSB)  Task  Force  on 
Department  of  Defense  In-House  Laboratories.  The  Sheingold  study  proposed  that 
individual  laboratories  be  reorganized  into  technical  centers,  each  possessing  a  "critical  mass" 
of  one  thousand  specialists  performing  R&D  as  well  as  demonstrating  the  workability  of 
prototypes.  About  70  percent  of  the  effort  would  be  devoted  to  in-house  activities  rather 
than  contract  monitoring,  and  each  center  director  would  control  the  resources  necessary  to 
accomplish  the  mission.  Within  days  of  the  formal  release  of  the  Sheingold  Report,  D  D  R&E 
directed  the  Navy  to  initiate  plans  to  establish  weapon  systems  development  centers.7 

Timing  was  excellent  for  staffing  of  the  Navy's  new  R&D  centers.  Science  and  technology 
funding  was  at  its  all-time  high,  partially  a  result  of  the  race  into  space  and  the  national  goal 
established  by  President  Kennedy  to  put  an  American  on  the  moon  by  the  end  of  the  decade. 
Added  to  those  lofty  and  noble  goals  of  national  interests,  space  exploration,  scientific 
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discovery,  and  contributing  to  the  interests  of  one's  country,  employment  by  a  military 
laboratory  held  the  promise  also  of  occupational  draft  deferments  for  a  growing  population  of 
baby-boom  male  scientists  and  engineers  (S&Es),  many  of  whom  did  not  relish  the  thought  of 
being  drafted  into  ground  combat  in  the  nation's  most  unpopular  war.  In  addition,  scientific 
employment  opportunities  took  a  drastic  downturn  at  the  end  of  the  decade,  and  new  M.S. 
and  Ph.D .  degree  scientists  offered  to  accept  Civil  Service  employment  at  entry  levels  as  low 
as  G S-5  instead  of  the  usual  G  S-9/ 11/ 12.  Thus  the  DOD  and  Naval  R&D  establishment 
were  able  to  increase  their  staff  easily,  picking  and  choosing  from  the  most  talented  of  young 
scientists  and  engineers.  The  centers  also  enhanced  the  education  levels  and  kept  morale  high 
through  formal  professional  training  programs  and  graduate  education,  freguently  provided 
on  site. 

The  Naval  Material  Command  proved  to  be  a  good  steward  of  the  Naval  R&D  centers  for  the 
nearly  two  decades  of  its  existence.  Even  though  S&T  funding  dropped  sharply  during  the 
latter  years  of  the  Viet  Nam  War  and  declined  at  a  pace  of  about  two  percent  per  year  for  the 
rest  of  the  decade  (and  since),  the  Naval  Laboratory  and  warfare  center  community  and 
associated  university  labs  developed  and  transitioned  the  reguisite  technologies  for  such 
advanced  systems  as  the  Aegis  Combat  System,  towed  arrays,  GPS,  laser-guided  projectiles, 
reactive  weapons,  and  directed  energy  technology,  to  name  only  a  few. 

However,  life  in  the  "DNL  Labs,"  as  the  community  was  known,  changed  dramatically  in 
April  1985  when  Navy  Secretary  John  Lehman  announced  the  disestablishment  of  the  Naval 
Material  Command,  beginning  a  period  of  instability  in  the  Naval  technical  community  that 
arguably  continues  to  this  day.  The  R&D  centers  and  the  university  laboratory  responsibilities 
were  placed  initially  under  the  0  ffice  of  Naval  Research  (0  NR),  but  0  NR  was  not  prepared 
to  manage  such  a  large  component  (over  20,000  people)  of  the  shore  establishment.  Less  than 
a  year  later  they  were  moved  to  the  Space  and  Naval  Warfare  Systems  Command  (SPAWAR), 
newly  created  to  oversee  systems  architecture.  Unfortunately  for  the  warfare  centers, 
laboratory  management  was  not  a  vital  interest  of  a  command  facing  the  difficult  task  of 
creating  the  Naval  System  for  Warfare  Systems  Architecture  and  Engineering.  Lurthermore, 
SPAWAR  was  placed  in  the  unenviable  position  of  coordinating  the  activities  of  warfare 
centers  that  received  most  of  their  funding  from  other  (peer,  but  larger)  Systems  Commands.8 

The  SPAWAR  laboratory  management  era  ended  in  January  1992,  when  the  office  of  the 
Director  of  Navy  Laboratories  was  formally  abolished.  Lour  new  "megacenters"  were 
established  [Air,  Control,  Command  (C2)  and  Ocean  Surveillance,  Surface,  and  Undersea], 
with  each  center  reporting  to  the  SYSCOM  representing  its  major  source  of  income.9  The 
movement  of  the  warfare  centers  under  the  N aval  Systems  Commands  was  the  most  visible  of 
three  major  events  in  the  early  1990s  that  fundamentally  changed  the  character  of  D  O  N  S&T 
execution.  This  move  occurred  despite  deep  concerns  within  the  R&D  community  that  the 
near-term  focus  necessary  for  the  SY  SCO  Ms  to  accomplish  their  primary  missions  would  lead 
to  a  slow  degradation  of  the  long-term  (S&T)  capabilities  of  the  warfare  centers. 

After  the  disestablishment  of  D  NL,  it  was  intended  that  two  new  committees  would  perform 
the  Laboratory/ Center  oversight  function.  The  Navy  Laboratory  and  Center  Oversight 
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Committee  (NLCOC),  chaired  by  the  Assistant  Secretary  of  the  Navy  for  Research, 
Development  and  Acquisition  [ASN  (RDA)]  and  including  the  Vice  Chief  of  Naval 
Operations  and  Assistant  Commandant  of  the  Marine  Corps  as  principal  members,  and  the 
Systems  Commanders  as  associate  members,  were  to  provide  policy  level  oversight.  The  last 
meeting  of  the  NLCOC  was  in  1992.  The  Navy  Laboratory/ Center  Coordinating  Group 
(NLCCG ),  which  includes  the  commanders  and  technical  directors  of  the  warfare  centers  and 
the  Commanding  Officer  and  Director  of  Research  of  NRL  (with  a  rotating  chair),  is  still 
active.  This  forum  is  effective  as  a  vehicle  for  communication  among  the  members  of  the 
community,  but  is  designed  to  focus  inward  rather  than  on  Naval  policy  and  external 
representation  issues. 

A  second  event  occurred  almost  unnoticed  in  1991,  but  which  had  a  significant  effect  on  the 
national  and  Naval  technology  base.  Public  Law  (PL)-914-41,  Section  203,  of  the  1971 
Military  Personnel  Authorization,  provided  for  private  industry  recovery  of  IR&D  expenses  as 
general  and  administrative  overhead,  for  building  the  future  business  and  technology  base. 
Companies  with  IR&D  programs  in  excess  of  $4  million  were  required:  (1)  to  submit  a 
technical  plan  describing  each  technical  project,  which  the  D  0  D  would  evaluate  for  "potential 
military  relevance;"  (2)  to  negotiate  an  agreement  with  the  D  0  D  which  established  an  IR&D 
ceiling  for  cost  recovery  (Note:  Most  companies  exceeded  the  cost  recovery  ceiling  each  year, 
with  company  dollars  paying  as  much  as  one-third  of  the  total  IR&D .);  and  (3)  to  present  an 
on-site  review  of  its  IR&D  program  to  the  DOD  at  least  once  every  three  years.10  In 
December  1991,  responding  to  pressure  from  industry  to  simplify  the  process  and  to  allow 
full  recovery  of  IR&D  expenses,  Congress  passed  PL-102-190,  which  stated  "that 
independent  research  and  development  and  bid  and  proposal  costs  of  D  0  D  contractors  shall 
be  allowable  as  indirect  costs  on  covered  contracts  to  the  extent  that  such  costs  are  allocable, 
reasonable,  and  not  otherwise  unallowable  by  law  or  under  the  Federal  Acquisition  (FAR)." 
The  legislation  allowed  full  recovery  of  IR&D  ecpenses  and  negated  the  requirement  for 
IR&D  ceiling  negotiations  with  DOD  (industry's  primary  goal  in  pursuing  the  legislation)  but 
also  negated  the  requirement  for  industry  reporting  and  DOD  review  and  oversight  of 
IR&D.11 

From  a  national  technology  base  perspective,  PL- 102-90  produced  two  unintended  negative 
consequences.  First,  it  allowed  (thus  effectively  encouraged)  industry  to  reduce  overhead  rates 
by  reducing  IR&D  investment.  IR&D  budgets  shrank  to  less  than  half  their  levels  before  the 
law  change,  and  the  character  of  IR&D  work  became  nearer-term  (more  like  bid  and  proposal 
funding)  as  the  Reagan  defense  budget  dropped  to  the  early  Clinton  administration  levels. 
Second,  the  DOD  and  industry  lost  overnight  a  forcing  function  to  encourage  and  assure 
access  by  individual  performers  to  each  other's  R&D,  ending  a  period  of  many  years  of 
mutually  advantageous  technical  communication  and  leverage  between  the  public  and  private 
sectors.  Ironically,  although  demands  for  leverage  of  private  sector  technology  increased 
steadily  throughout  the  decade  following  passage  of  PL-102-90,  private  sector  technology 
development  became,  with  few  exceptions,  far  less  visible  to  the  D  0  D . 

The  third  event  was  the  loss  of  the  6.2  Laboratory  Independent  Exploratory  Development 
(IED )  Program  in  1993,  cancelled  because  of  D  0  N  S&T  reductions  and  a  historically  difficult 
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Congressional  budget  defense  (a  casualty  of  Congressional  staff  concerns  over  after-the-fact 
program  review  and  oversight).  IED  was  a  relatively  small  account  (nominally  $25 
million/  year),  with  funding  allocated  to  NRL  and  the  warfare  centers  to  complement  the  6.1 
Laboratory  Independent  Research  Program  of  similar  size,  which  still  exists  today.  These 
programs  represented  together  an  exceptionally  effective  DO  N  investment  in  new,  generally 
high-risk/  high-payoff  technologies  -  "disruptive  technologies,"  in  the  terminology  of  Bower 
and  Christenson,12  that  has  become  a  current  term  of  choice.  IED  funding  at  a  specific 
Division  site  was  modest  [e.g.  an  activity  the  size  of  Naval  Surface  Warfare  Center's  (NSWC) 
Carderock  Division  might  receive  $2  million  each  of  Independent  Research  and  IED  funds], 
and  frequently  was  divided  into  increments  of  less  than  one  work-year  each.  Selection  of 
projects  for  In-house  Laboratory  Independent  Research  (ILIR)  or  IED  funding  gave  NRL 
and  the  warfare  centers  an  excellent  way  to  reward  innovation  and  to  encourage  the  best  and 
brightest.  Technologies  such  as  GPS,  reactive  weapons,  and  submarine  bow  planes  were 
supported  in  their  infancy  with  IED  funds.  The  loss  of  IED  represented  a  major  blow  to  the 
S&T  communities  within  the  centers,  forcing  many  scientists  and  engineers  into  non- S&T 
career  paths. 

After  the  cancellation  of  IE  D ,  0  NR  created  and  maintained  a  similar  Applied  Research  "Base 
Program"  at  the  Naval  Research  Laboratory.  The  concept  proved  to  be  an  important 
resource  for  the  NRL  Director  of  Research  to  assure  future  excellence  in  research  and  to 
reward  innovation.  Base  Program  funding  has  been  Congressionally  defensible  (unlike  IE  D , 
for  which  defense  was  difficult),  but  S&T  base  programs  were  not  established  at  the  four 
major  warfare  centers. 

In  1995,  the  Naval  Air  Warfare  Center  (NAWC)  headquarters  staff  was  disestablished,  and  the 
R&D  and  S&T  activities  remaining  from  what  was  known  in  1985  as  the  Naval  Weapons 
Center  and  the  Naval  Air  Development  Center  were  integrated  into  the  NAVAIR  structure. 
Three  years  later  the  components  of  the  DON  Command,  Control,  Communications, 
Computer,  and  Intelligence  (C4I)  R&D  infrastructure  were  integrated  into  SPAWAR. 
Although  NAVSEA  attempted  to  integrate  the  NSWC  components  into  its  structure  in  1999 
(but  maintained  the  warfare  center  identity  and  the  T echnical  D  irector  position),  at  the  time  of 
this  writing  both  NSWC  and  the  Naval  Undersea  Warfare  Center  (NUWC)  retain  their  status 
as  independent  entities  within  the  Command. 

The  Marine  Corps  became  a  new  player  within  the  D  0  D  laboratory  system  in  1995  when  the 
Commandant  established  the  Marine  Corps  Warfighting  Laboratory  (MCWL)  at  Quantico, 
Virginia.  Through  a  series  of  major  exercises  and  warfighting  experiments,  MCWL 
introduced  the  first  "Battle  Lab"  of  the  D  epartment  of  the  Navy.  In  2001,  MCWL  established 
its  first  civilian  Technical  Director  position,  to  make  the  organization  more  compatible  with 
the  Naval  RDT&E  infrastructure  and  to  ensure  corporate  memory. 

Thus  the  Department  of  the  Navy  entered  the  new  millennium  with  a  SY SCO M- centric 
warfare  center  infrastructure  and  a  new  and  growing  Marine  Corps  R&D  presence.  The 
current  organizational  structure  is  arguably  better  coupled  to  the  acquisition  and  budget 
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processes  than  in  the  old  "DNL  Lab"  days.  There  are  potentially  serious  near-term  issues, 
however,  which  include: 

1.  An  aging  DON  senior  work  force,  with  an  entire  generation  eligible  to  retire  within 
five  years  and  potentially  without  trained  replacements  in  the  pipeline  to  carry  forward; 

2 .  AN  aval  S&T  budget  that  has  declined  steadily  since  the  1 960s,  specifically  in  Applied 
Research; 

3.  Declining  industry  IR&D  investment,  now  focused  primarily  on  the  short-term  (a 
complement,  but  not  an  alternative  to  D  0  D  S&T);  and 

4.  The  requirement  for  a  strategic  vision  for  the  DON  RDT&E  infrastructure  and 
resources  to  focus  technology  investment  and  assure  technological  superiority  for  the 
Navy- After- Next. 
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III.  DOD  Laboratory  and  Warfare  Center  Studies 


There  has  been  no  shortage  of  studies  of  the  D  0  D  laboratory  infrastructure  over  the  past  two 
decades.  Indeed,  the  greatest  difficulty  in  examining  the  health  of  the  system  is  in  filtering  and 
distilling  the  information  overload  into  useful  and  objective  conclusions.  The  following 
discussion  highlights  some  studies  of  more  prominence  or  greater  depth,  focusing  in  particular 
on  the  reports  of  the  D  SB  of  the  past  decade.  Although  only  a  small  sample  of  the  available 
information  is  highlighted,  the  degree  of  consistency  across  all  the  studies  is  exceptionally 
high,  including  even  those  studies  conducted  long  before  the  Reagan  defense  buildup  of  the 
1980s.  Unfortunately,  few  studies  were  conducted  using  formal  metrics  or  a  scientific 
approach.  Many  began  with  the  basic  assumption  that  DOD  laboratories  were  essential,  and 
neither  investigated  nor  attempted  to  justify  their  appropriate  role  within  the  D  0  D  and  the 
national  system.  Reports  that  provide  significant  justification  for  the  existence  of  the 
laboratory  system  are  specifically  highlighted  below.  The  comments  included  below  are  not 
intended  to  represent  all  the  findings  of  the  various  studies,  but  to  provide  a  sampling  of 
significant  comments  as  related  to  the  laboratory  and  warfare  center  infrastructure.  G  reater 
detail  is  included  for  the  most  recent  studies.  For  the  sake  of  brevity,  common  findings 
(which  are  legion)  may  not  be  repeated.  Significant  studies  include: 

"Report  of  the  Defense  Science  Board  Task  Force  on  Efficient  Utilization  of  Defense 
Laboratories,"  October  2000.13  This  report  responds  to  Congressional  reporting 
reguirements  contained  in  Section  913  of  the  FY  -2000  National  D  efense  Authorization  Act 
(See  Appendix  A). 

Findings: 

1 .  The  military  laboratories  represent  an  important  source  of  innovation. 

2.  The  current  Civil  Service  salary,  regulations  make  recruiting  and  maintaining  guality 
professional  staff  extremely  difficult. 

3 .  T  he  pro  cess  and  regulatio  ns  fo  r  approval  o  f  Senio  r  E  xecutive  Service  appo  intees  make 
it  extremely  difficult  to  recruit  the  leading  technical  and  administrative  professionals. 

4.  D  0  D  lab  personnel  decreased  36  percent  in  last  ten  years;  many  of  the  younger  staff 
members  with  training  in  current  technology  are  no  longer  employed. 

5.  About  30  percent  of  the  professional  staff  are  within  five  years  of  retirement 

6.  Many  of  the  D  O  D  laboratories  are  physically  separated  from  the  development  centers 
that  are  the  natural  recipients  of  their  innovations,  impeding  transfer  of  technology. 

7.  Based  on  private  sector  experience,  competition  between  defense  laboratories  could 
enhance  productivity. 

Recommendations: 

1 .  The  Secretary  of  D  efense  (SE  CD  E  F)  should  instruct  the  Services  to  bring  in  private 
sector  S&Es  under  contracts  and  IPA  agreements  from  universities,  industry  and  non¬ 
profits  for  a  majority  of  the  laboratory  professional  staffs. 

2.  SECDEF  should  seek  relief  from  procedures  inhibiting  Executive  Service 
appointments 

3.  If  consolidations  are  undertaken,  locate  laboratory  operations  physically  close  to 
Service  development  and  procurement  centers. 
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4.  The  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  [USD 
(AT &L)]  should  maintain  competition  between  defense  laboratories,  especially  for  6.1 
Basic  Research. 

5.  D  D  R&E  should  utilize  university  technology  historians  to  develop  understanding  of 
past  relevance  of  defense  laboratories. 

6.  DDR&E  should  utilize  the  National  Academies  of  Science  (NAS)  to  examine 
periodically  (e.g.  every  three  years)  the  quality  of  the  defense  laboratories. 

"The  Defense  Science  Board  Letter  Report  on  DOD  Science  and  Technology 
Program/'  August  200014  and  "Report  of  the  Defense  Science  Board  Task  Force  on 
Defense  Science  and  Technology  Base  for  the  21l  Century/'  June  1998.15  This  letter 
report  responds  to  the  request  of  Section  212  of  the  FY  2000  D  efense  Appropriation  Report 
that  the  D SB  provide  an  assessment  regarding  the  appropriate  funding  level  for  the  DOD 
Science  and  T echnology  Program. 

Findings: 

1 .  "DOD  should  be  requesting  higher  levels  of  funding  for  the  S&T  Program." 

2.  "Much  of  current  industrial  research  has  a  very  short  time  horizon  and,  in  addition, 
tends  to  be  focused  on  incremental  improvements  of  current  civilian  products." 

3.  The  current  Civil  Service  Personnel  System  has  a  very  negative  impact  on  the 
capabilities  and  morale  of  the  DOD  and  Service  Laboratory  and  Center  technical 
personnel." 

4.  The  focus  of  the  current  DOD  S&T  Program  is  primarily  on  incremental 
improvements  in  current  capabilities." 

5.  DOD  and  Service  research  facilities  are  physically  disconnected  from  weapon 
development  and  procurement  organizations,  and  many  are  old  and  poorly  equipped. 

Recommendations: 

1 .  The  D  0  D  S&T  budget  should  be  increased  by  $1  billion/  year. 

2.  "DOD  should  not  depend  on  civil  sector  research  for  all  its  needs." 

3.  DOD  should  depend  less  on  the  Civil  Service  System  and  more  on  universities  and 
industry  to  provide  professional  personnel  for  D  0  D  and  Service  laboratories. 

4.  DOD  should  emphasize  innovative  technology  initiatives  leading  to  new  military 
capabilities. 

5.  DOD  and  the  Services  should  consolidate  and  modernize  their  R&D  facilities. 

"Report  of  the  Defense  Science  Board  Task  Force  on  the  T echnology  Capabilities  of 
Non-DOD  Providers,"  June  2000.16  This  Task  Force  was  asked  to  support  the 
D  epartment's  in-house  study  by  examining  the  capabilities  of  non-D  0  D  providers  to  provide 
warfighting  technology,  engineering  and  test  capabilities  for  future  U.S.  military  forces. 

Findings: 

1 .  Commercial  industry  is  generally  disinterested  in  developing  products  for  D  0  D ,  as  the 
opportunity  costs  are  too  great,  cost  of  doing  business  is  too  high,  etc. 

2.  The  U.S.  and  worldwide  research  base  has  increased  significantly  since  WWII,  but  the 
DOD  contribution  has  remained  essentially  constant. 

3.  DOD/  Service  S&T  and  acquisition  infrastmcture  has  decreased  25  percent  since 
1990;  funding  has  decreased  35  percent. 
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4.  "D  0  D  should  concentrate  the  efforts  of  its  limited  laboratory  and  center  resources  on 
those  unique  military  technologies  and  systems  which  are  of  crucial  importance  to 
maintaining  U.S.  military  preeminence." 

5.  "...Commercial  firms  today  have  greatly  reduced  their  funding  of  long-term,  generic 
technology  development,  meaning  technology  that  may  not  find  applications  until  ten 
years  and  beyond." 

Recommendations: 

1 .  E  stablish  the  0  ffice  of  G  lobal  T echnology  Acquisition,  funded  at  $100  million/  year 
and  focusing  on  militarily  important  technologies,  "disruptive  technologies,"  novel 
approaches  to  acquire  technology,  and  commercial  contracting/  licensing  practices. 

2.  Increase  to  50  percent  the  Service  S&T  management  and  laboratory  staff  contracted 
from  the  private  sector. 

3 .  I  ncrease  funding  o  f  D  0  D  -  relevant  university  research  by  30  percent. 

4.  Initiate  a  high-level  "Packard-like"  Commission  to  develop  an  integrated 
requirements/  acquisition  process. 


Defense  Science  Board.  "Human  Resources  Strategy/'  February  2000.17 

Findings: 

1 .  Service  laboratories  downsized  35  percent  during  the  thel990s. 

2.  Thirty  percent  of  the  D  0  D  laboratory  staff  is  within  five  years  of  retirement,  with  no 
replacement  staff  in  training  to  provide  continuity  of  technical  expertise  and  corporate 
memory. 

3.  The  staff  and  capability  losses  have  potentially  serious  negative  impacts  on  future  military 
capabilities,  i.e.  can  the  U.S.  achieve  a  Revolution  in  Military  Affairs  before  an  adversary? 

4.  The  report  agrees  strongly  with  past  D  SB  studies  that  the  inflexible  Civil  Service 
system  is  to  blame  for  many  of  the  problems. 

Recommendations: 

1 .  D  0  D  should  acquire  a  high-quality  staff  from  universities,  non-profits  and  industry. 

2.  Reinstate  PL-80-313  to  enhance  outside  recruiting. 

3.  Civil  Service  post-employment  and  pension  restrictions  should  be  relaxed. 

Defense  Science  Board.  "FFRDCs  and  UARCs  Independent  Advisory  T ask  Force," 

January  199718  and  "Task  Force  on  the  Role  of  FFRDCs  in  the  Mission  of  the  DOD," 

April  1995.19 

Findings  and  Recommendations: 

1.  In-house  laboratories  and  private  industry  have  gained  some  of  the  unique 
competence  that  initially  characterized  FFRD  Cs. 

2.  Reliance  of  D  0  D  on  FFRD  Cs  and  UARCs  was  reaffirmed  for  well-defined  functions. 

3.  DOD  should  accurately  define  FFRD  C/  UARC  tasking. 

4.  DOD  Should  use  open  competition  when  appropriate. 
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"Navy  Laboratory  and  Center  Coordinating  Group  (NLCCG)  Ad  Hoc  Study  of  the 
Value  of  N aval  Warfare  Centers,"  N ovember  1996.20 
Addressed  need  for  in-house  laboratory  system. 

Findings  and  Recommendations: 

Naval  in-house  technical  organizations  exist  to  ensure  that: 

1 .  T echnical  program  risk  is  recognized  and  minimized. 

2.  DON  receives  maximum  benefit  from  technology  and  other  modernization 
investments. 

3.  Warfighters  are  provided  rapid,  direct  technical  support. 

4.  DON  maintains  a  technical  corporate  memory. 

5.  A  long-term  view  is  available  for  decision-making. 

6.  Special  facilities  are  maintained. 

7.  A  technical  honest  broker  is  available  to  D  0  N . 

8.  Stewardship  is  maintained  across  the  life  cycle. 

N aval  Research  Advisory  Committee.  "N aval  Research  and  Development,"  October 

1994.21 

Findings  and  Recommendations: 

1.  There  are  significant  organizational  weaknesses  in  Naval  R&D  structure,  personnel 
policies,  financial  execution  and  planning. 

2.  Process  weaknesses  include  weak  integration  between  Naval  doctrine  and  R&D,  a 
reguirements  process  that  does  not  address  prioritization  or  assess  cost,  risk  and 
affordability  (Note:  The  current  Future  Naval  Capabilities  (FNC)  process  was 
established  to  address  precisely  these  concerns;  applied  only  to  the  Naval  S&T 
Program.). 

3.  Navy  must  protect  and  retain  critical  D  0  N-unigue  resources  (products,  capabilities, 
facilities,  personnel)  before  the  ability  is  lost. 

4.  Navy  should  create  a  single  Warfare  Systems  Command  as  the  focal  point  for  the 
Navy's  R&D  and  material  needs. 

White  House  Report  94:  "OSD  Interagency  Review  of  Federal  Labs"  of  12  September 

1994.22 

Addressed  need  for  in-house  laboratory  system. 

Findings  and  Recommendations: 

D  0  D  in-house  laboratory  system  is  needed: 

1 .  To  ensure  lowest  cost  to  the  sponsor; 

2.  To  avoid  technological  surprise; 

3.  To  provide  rapid,  flexible  responses  to  military  needs; 

4.  To  conduct  inherently  governmental  tasks; 

5 .  To  maintain  technical  co rpo rate  memo ry; 

6.  To  provide  technology  and  system  integration; 

7.  To  provide  continuity  across  system  life  cycles;  and 

8.  To  ensure  long-term  military  R&D . 
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Defense  Science  Board.  "Task  Force  of  Defense  Laboratory  Management'"  April 

1994.23 

Findings  and  Recommendations: 

1.  "Resize  and  restructure  the  Defense  Laboratories  to  enable  them  to  respond  to 
external  changes  and  better  use  their  resources." 

2.  "Pursue  a  vigorous  program  of  outsourcing  defense  laboratory  activities." 

3.  E  stablish  long-term  managerial  oversight  by  D  D  R&E . 

Defense  Science  Board.  "Defense  Technology  Strategies/'  January  1992.24 

Findings  and  Recommendations: 

1.  The  military  benefited  greatly  in  the  Gulf  War  from  DOD  laboratory  technology 
prototypes  deployed  rapidly  into  the  field. 

2.  DOD  should  make  maximum  use  of  Advanced  T echnology  D  emonstrations  (ATD  s) 
and  Advanced  Concept  Technology  Demonstrations  (ACTDs)  for  rapid  transition  of 
technology  into  military  use. 

Adolph  Commission:  Federal  Advisory  Commission  Report  on  "Consolidation  and 

Conversion  of  Defense  R&D  Labs,"  30  September  199L25 

Addressed  need  for  in-house  laboratory  system. 

Findings  and  Recommendations: 

The  DOD  laboratory  system  is  needed: 

1 .  To  allow  the  Services  to  be  smart  buyers  of  new  and  improved  systems; 

2.  To  act  as  principal  agents  for  maintaining  the  technology  base; 

3.  To  ensure  technological  innovation  and  avoid  technological  surprise; 

4.  To  support  acguisition  programs; 

5.  To  provide  special  facilities  not  practical  in  the  private  sector; 

6.  To  provide  rapid  response  in  times  of  crisis; 

7.  To  provide  constructive  technical  advice; 

8.  To  support  the  user  in  application  of  new  technology; 

9.  To  translate  user  needs  into  technology  reguirements  for  industry;  and 

10.  To  serve  as  an  S&T  training  ground  for  civilian  and  military  acguisition  personnel. 

Defense  Science  Board.  "R&D  Strategy  for  the  1990s,"  October  1990.26 

Findings  and  Recommendations: 

1.  D  SB  expressed  serious  concerns  with  deteriorating  guality  of  personnel.  "We  must 
begin  now  to  develop  a  set  of  activities  that  will  represent  our  best  efforts  ...  [to 
develop]  inducements  for  attracting,  training  and  retaining  the  best  people  to  serve  our 
country." 

2.  Increase  centralization  of  R&D  management  within  DOD.  (Note:  This 
recommendation  was  perceived  by  the  Services  as  a  potential  threat  to  their  laboratory 
systems  and  was  a  factor  in  the  Services'  consolidation  of  their  RDT&E  infrastructure 
into  "megacenters.") 

3.  DOD  should  exploit  private  sector  R&D . 

4.  DOD  must  maintain  the  ability  to  generate  breakthrough  technology. 
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Defense  Science  Board.  "Report  on  T echnology  Transition  in  the  DOD;"  December 
1988.27 

Findings  and  Recommendations: 

1.  Navy  ATD  Program  embodied  intent  of  Packard  Commission  recommendations  on 
prototyping  of  defense  technologies. 

2.  ATD  s  encourage  technology  transition. 

3.  All  Services  should  apply  at  least  50  percent  of  their  6.3  budget  to  ATD  s. 

CNO  Executive  Board  Study,  published  in  U .S.  N aval  Institute  Proceedings,  "Eating 
Our  Seed  Com,"  by  Kaii  M.  Duff,  July  1984.28 

Findings  and  Recommendations: 

1.  The  DOD's  "fly  before  buy"  policy  produced  severe  adverse  impact  on  RDT&E 
accounts,  as  no  fiscal  adjustments  made  to  accommodate  greatly  increased  costs. 

2.  Result  is  dramatic  S&T  decline  as  percentage  of  budget,  inadequate  6.3  prototype 
funding,  and  acquisition  program  stretch-out. 

3.  Chronic  6.3  program  instability  must  be  rectified  to  stabilize  acquisition  program 
costs. 

4.  "A  policy  of  structured  RDT&E  growth  for  competitive  experimental  prototyping  is 
necessary  and  is  the  most  feasible  means  of  introducing  corrective  resource  allocation 
to  the  Navy's  long-term  operational  capabilities  and  acquisition  process." 

"N aval  Material  Command  Mission  Review  Panel  Report,"  13  December  1982. 
Addressed  need  for  in-house  laboratory  system.29 
Findings  and  Recommendations: 

Naval  laboratories  are  needed: 

1 .  To  help  the  N avy  be  a  smart  buyer; 

2.  To  ensure  technical  leadership  and  expertise; 

3.  To  provide  warfare  and  system  analyses; 

4.  To  interact  with  and  support  the  operating  forces; 

5.  To  plan  and  conduct  the  D  0  N  S&T  program; 

6.  To  provide  an  independent  source  of  prototyping,  concept  evaluation,  systems 
engineering  and  concept  validation; 

7.  To  ensure  technology  transition; 

8.  To  provide  quick  response  to  crises;  and 

9.  To  provide  special  RD  T &E  facilities. 

Under  Secretary  of  D  efense  for  Research  and  E  ngineering.  "An  Independent  Review 
of  DOD  Laboratories,"  March  1982.30 
Findings  and  Recommendations: 

1 .  There  is  a  significant  "loss  of  work  force  competence"  in  the  D  0  D  laboratory  system. 

2.  There  is  "extensive  erosion  in  the  competence,  morale,  and  effectiveness  of  this  work 
force  [which]  if  continued,  would  lead  to  the  decline  of  the  laboratory  system." 
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Under  Secretary  of  Defense  for  Research  and  Engineering  Report,  "Reguired  In- 
H  ouse  C  apabilities  for  D  0  D  RD  T  & E 10  ctober  1980.31 
Addressed  need  forDOD  in-house  laboratory  system. 

Findings  and  Recommendations: 

The  D  0  D  laboratories  are  needed: 

1 .  To  help  the  D  0  D  be  a  smart  buyer; 

2 .  To  manage  RD  T  &E  programs; 

3.  To  provide  technical  intelligence  assessments; 

4.  To  provide  options  for  future  systems; 

5.  To  exploit  new  technological  opportunities; 

6.  To  interact  with  and  understand  the  military  user;  and 

7.  To  evaluate  contractor  proposals  and  products. 

White  House  Report  79:  Federal  Coordinating  Council  for  Science,  Engineering  and 
Technology.  "Application  of  OMB  Circular  A-76  to  R&D:  an  R&D  Management 
Approach,"  31  October  1979.32 
Addressed  need  forDOD  in-house  laboratory  system. 

Findings  and  Recommendations: 

The  D  0  D  laboratories  are  needed: 

1 .  To  help  the  D  0  D  be  a  smart  buyer; 

2.  To  conduct  mission-oriented  studies,  technical  analyses  and  evaluation; 

3.  To  provide  R&D  for  the  long  term; 

4.  To  provide  independent  T est  and  E  valuation  (T &E ); 

5.  To  ensure  R&D  corporate  memory; 

6.  To  provide  rapid  response  capability;  and 

7.  To  support  unigue  R&D  facilities  not  feasible  in  the  private  sector. 

There  is  a  remarkable  degree  of  consistency  in  the  reports  listed  above  and  in  the  many 
dozens  more  that  could  have  been  referenced.  This  is  true  whether  the  studies  focused  on  the 
value  of  the  laboratory  system  or  focused  on  the  problem  areas. 

Points  of  universal  agreement  with  respect  to  the  need  for  an  independent  in-house  RDT&E 
capability  in  D  0  D  include: 

•  To  help  D  0  D  and  the  Services  to  be  smart  buyers; 

•  To  ensure  the  ability  to  respond  rapidly  in  times  of  crisis; 

•  To  support  facilities  and  capabilities  not  practical  in  the  private  sector; 

•  To  execute  S&T  programs  and  maintain  the  long  term  view; 

•  To  provide  direct  technical  support  to  the  warfighter;  and 

•  To  maintain  technical  co rpo rate  memo ry. 

Almost  all  of  the  reports  recognized  that  the  laboratory  hiring  slowdowns/  freezes  of  the 
1970s  and  1980s  after  several  years  of  significant  "staffing  up"  created  an  entirely  predictable 
future  loss  of  technical  competence  and  technological  leadership.  Such  an  outcome  was 
forecast  as  early  as  the  1982  articles  referenced  above.  T aken  as  a  whole,  the  dominant  points 
of  concern  include: 
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•  The  DOD  laboratory  system  has  declined  in  numbers  by  approximately  35  percent 
since  1990. 

•  The  DOD  laboratory  system  is  likely  to  lose  by  retirement  within  five  years  a 
generation  of  experienced  and  talented  staff,  with  no  trained  backup  to  carry  forward. 

•  Civil  Service  hiring  practices  and  policies  severely  limit  hiring  of  the  best  personnel, 
either  in  career  or  appointee  status. 

•  S&T  funding  decline  compounds  the  problem  by  inhibiting  incentives  for  recruitment 
of  the  "best  and  brightest,"  raising  acguisition  costs  and  increasing  risk  for  the  future. 

The  dominant  recommendations  include: 

•  DOD  must  augment  its  senior  staff  through  contract  staff  and  IPAs  from  industry, 
academia  and  non-profits. 

•  DOD  must  utilize  and  exploit  commercial  technology  wherever  and  whenever 
possible. 

•  DOD  must  act  guickly. 
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IV.  Metrics  for  Evaluation  of 

Research  and  Development  -  Sampling  of  Previous  Studies 

Just  as  there  is  no  shortage  of  studies  on  the  DOD  RDT&E  infrastructure,  there  is  no 
shortage  of  articles  on  study  metrics.  While  there  is  less  unanimity  in  the  collective 
recommendations  than  for  the  laboratory  studies,  certain  specific  patterns  emerge,  particularly 
with  respect  for  metrics  to  evaluate  R&D .  A  sampling  is  listed  below: 

Army  Research  Laboratory  T echnical  Assessment  Board,  1999. 33 

Conclusions:  The  outcome  of  R&D  activity  often  cannot  be  guantified  in  advance.  In  fact, 

it  may  lag  the  initial  R&D  by  several  decades. 

Recommendations:  Recommended  a  combination  of  peer  review,  customer  evaluation,  and 
performance  measures  (papers,  patents,  backlog,  diversity,  cycle  time,  etc.). 

Brown  and  Svenson,  "Measuring  R&D  Productivity,"  Industrial  Research  Institute, 
1998.34 

Conclusions:  Far  too  much  emphasis  on  in-process  measurement  and  feedback,  behavior, 
R&D  outputs  (papers,  presentations,  patents,  awards,  etc.) 

Recommendations: 

1 .  Focus  on  external,  not  internal  measures. 

2.  Focus  on  outcomes  (quality,  guantity,  cost),  not  behavior. 

3.  Measure  only  valuable  accomplishments. 

4.  Make  the  system  simple  -  no  more  than  six  to  eight  key  indices. 

5.  Make  the  system  more  objective  than  subjective  (difficult  for  R&D ). 

GAO  Report,  "Measuring  Performance  -  Strengths  and  Limitations  of  Research 
Indicators,"  March  1997.35 

Conclusions:  "...  many  of  the  R&D  output  measures  tracked  by  the  private  sector  do  not 
apply  directly  to  the  federal  government." 

Recommendations:  Recommends  combination  of  return  on  investment  (ROI)  evaluations, 
patents,  biblio  metrics,  and  peer  review. 

Adolph  Commission,  Federal  Advisory  Commission  Report  on  "Consolidation  and 
Conversion  of  Defense  R&D  Labs,"  30  September  199L36 

Conclusions:  A  single  set  of  measures  of  effectiveness  (MOEs)  will  not  apply  across  the 
board  to  every  laboratory. 

Recommendations:  Recommended  set  of  input  and  output  MOEs  for  DOD  laboratory 
reviews. 

Input  MOEs: 

1.  Mission  -  clear,  substantive,  unigue 

2 .  Types  and  percentages  o  f  funding  and  applied  wo  rk-years 

3.  Specific  tasking/  responsibilities 

4.  Workload  -  demonstrated  need  for  products 

5.  Quality  of  facilities 

6.  E  mployee  credentials  -  degrees,  G  PA,  training 
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Output  MO Es  (applied  as  appropriate): 

1 .  T  ransitio  ns  to  warfare  systems 

2.  Reports/  presentations  -  internal/ external 

3.  New  system  concepts  -  proposed/  successful 

4.  Proposals  of  all  kinds  -  proposed/  successful 

5.  Systems  specifications  developed 

6.  Systems  analysis 

7.  Invention  disclosures,  patents 

8.  Peer  review  -  qualitative 

9.  External  awards 

10.  Reputation  among  users  -  commendations,  surveys 

11.  Audits  -  types  of  problems,  resolved  vs.  unresolved 

N  ational  Research  Council  (Lawrence)  Study  of  Feasibility  of  Assessing  the  Quality  of 
S&Es  in  DOD  Laboratories,  March,  1985.37 

Conclusions:  "Identical  quality  measures  cannot  reasonably  be  applied  to  all  DOD 
laboratory  S&Es."  For  example,  a  famous  scientist  who  publishes  regularly  in  respected 
journals  and  a  relatively  unknown  scientist  who  develops  technology  to  lower  the  cost  of  a 
major  weapons  system  may  be  judged  very  differently  by  peer  and  customer  communities. 
Recommendations:  Measures  may  include  a  subset  of:  (1)  bibliometry;  (2)  technical  reports; 
(3)  awards;  (4)  invitations  to  scientific  meetings;  (5)  peer  review;  and  (6)  supervisory  ratings. 

Army  Science  Board  Summer  Study,  1982.38 

Conclusions:  "There  is  no  objective  basis  for  concluding  that  the  quality  [of  S&Es]  is 
adequate  to  meet  requirements... " 

Recommendations:  Focus  on  quality,  not  quantity;  discontinue  practice  of  forcing  S&Es 
into  management  before  promotion  to  senior  ranks. 
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V.  Metricsfor  Evaluation  of 
Research  and  Development  -  Recommendations 

The  common  themes  in  the  above  references  are  clear,  that  objective  evaluation  of  R&D  is 
difficult  at  best,  and  that  one  size  does  not  fit  all  in  terms  of  R&D  metrics.  Another 
consistent  theme,  which  meets  the  criterion  of  common  sense,  is  that  the  number  of  metrics 
for  an  evaluation  should  be  small,  under  ten  and  preferably  no  more  than  a  half-dozen.  With 
these  facts  and  assumptions  in  mind,  and  after  considerable  discussion  with  experts  in  the 
DOD  R&D  community,  the  following  metrics  are  suggested  for  a  relatively  objective 
evaluation  of  the  health  of  science  and  technology  in  the  D  0  D  laboratory  and  warfare  center 
system.  Because  "one  size  does  not  fit  all"  for  R&D  assessment,  other  metrics  may  be 
applicable  on  an  activity- specific  basis. 

L  Quality  and  military  impact  of  the  product.  While  most  would  agree  that  guality  and 
excellence  in  the  final  product  is  the  ultimate  standard  against  which  any  activity  should  be 
judged,  it  is  a  very  difficult  metric  to  utilize,  especially  for  S&T  output.  Customer  satisfaction 
is  a  subjective  but  essential  factor  in  the  evaluation,  and  guality  can  be  freguently  an  "eye  of 
the  beholder"  characteristic. 

Recommended  evaluation  criteria  include: 

1 .  T ransitions  to  acguisition  programs  and  to  the  fleet  (for  6.2/  6.3); 

2.  Problems  solved  (or  avoided)  by  the  organization; 

3.  Rapid  response  capability  in  crisis; 

4.  Smart  buyer  functions,  savings  generated  and  costs  avoided;  and 

5.  Perceived  responsiveness  and  support  to  the  military  user. 

Data  freguently  would  be  subjective  (although  existence  of  a  product  in  a  fielded  system  is 
clearly  a  success),  and  would  be  gathered  from  on-site  assessments  of  the  current  warfare 
center  programs,  customer  interviews/  surveys,  Service  databases,  and  from  the  activities 
themselves.  Quality  of  the  data  available  immediately  is  poor-to-fair,  with  insufficient 
granularity  for  simple  analysis. 

2.  Core  Competencies.  This  metric  is  very  important  in  a  time  of  potential  funding 
reductions,  future  downsizing  and /  or  Base  Closure  and  Realignment  Commission  (BRAC) 
actions.  While  the  first  metric  addresses  the  guality  of  the  product  of  the  organization  for 
DOD  application,  the  core  competency  metric  focuses  more  on  the  internal  capabilities  that 
make  the  organization  uniguely  valuable  to  the  D  0  D .  Assessments  would  be  conducted  in  a 
manner  consistent  with  the  seminal  articles  on  the  subject  by  G .  Hamel  (U.  of  Michigan)  and 
C.  K.  Prahalad  (London  Business  School)  and  published  in  the  Harvard  Business  Review  in 
the  1989-91  time  frame.  Hamel  and  Prahalad  introduced  the  twin  concepts  of  strategic  intent 
(aggressively  stretching  resources  rather  than  fitting  resources)  and  core  competence 
(collective  learning  in  an  organization,  especially  with  respect  to  integration  diverse  skills  and 
multiple  streams  of  technologies).  By  their  model,  the  core  competence  list  of  an  activity 
would  be  very  short,  fully  understood  and  fully  supported  by  all  levels  of  the  organization  (e.g. 
Honda  -  best  automotive  engines  and  engineering).39,40 
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Recommended  evaluation  criteria  include: 

1 .  E ssential  competencies  existing  only  at  the  specific  activity  -  must  maintain. 

2.  E  ssential  competencies  existing  also  at  other  D  0  D  or  private  sector  activities 
-  maintain,  share  or  hand  off  to  other  activity. 

3.  Identify  competencies/ functions  that  may  not  be  essential,  potential  to 
discontinue. 

In  addition  to  site  visits,  data  would  be  gathered  from  the  Services,  data  calls,  IR&D  database, 
and  industry  associations.  With  few  exceptions,  relevant  high  guality  data  is  available. 

3.  Value-added  of  the  organization.  The  intent  of  this  metric  is  to  determine  whether 
organization  performs  vital  and  unigue  -  and /  or  uniguely  governmental  -  functions,  functions 
that  cannot  or  should  not  be  performed  elsewhere.  Also,  is  the  activity  an  integral  part  of  the 
decision-making  process  or  simply  a  gatherer/  compiler  of  information  for  the  decision-maker. 

Recommended  evaluation  criteria  include: 

1.  Level  of  S&T  conducted  in-house,  contracted. 

2.  Level  of  development  conducted  in-house,  contracted. 

3.  Level  of  other  hands-on  work  vs.  contract  monitoring? 

4.  Does  activity  determine  the  cause  of  problems  (regardless  of  who  works  on 
problem)?  Validate  correctness  of  solutions?  Conduct  final  certification  of 
systems? 

5.  How  does  the  activity  help  D  0  D  to  be  a  smart  buyer? 

6 .  0  ther  inherently  governmental  functio  ns. 

In  addition  to  on-site  visits,  data  generally  would  be  available  from  the  Services,  augmented  by 
a  data  call  to  the  centers  to  address  the  guestions.  Q  uality  of  the  data  available  immediately  is 
fair  to  good. 

4.  Quality  of  the  technical  staff.  This  metric  is  especially  important  with  respect  to 
concerns  that  a  generation  of  expertise  may  retire  soon,  leaving  behind  a  generation  gap  of 
capability.  The  issues  are  not  simply  demographics  and  hiring  trends,  but  also  in  terms  of  the 
character  of  the  work  and  training  for  the  new  hires,  i.e.  are  they  conducting  S&T,  managing 
contracts,  developing  budgets,  etc.? 

Recommended  evaluation  criteria  include: 

1 .  D  emographics  of  technical  personnel,  generation  gaps  of  expertise. 

2.  Maintenance  of  the  technical  corporate  memory. 

•  Current/  projected  technical  capabilities 

•  Current/  projected  technical  deficiencies. 

3.  Hiring  trends. 

•  Age  of  new  hires 

•  Education  level 

•  Education  mix 

4.  Character  of  work/  training  for  new  hires. 

•  S&T 
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•  Hands-on  R&D 

•  Project/  program  management 

•  Contract  management 

•  In-service  engineering 

•  FY  D  P/  budget  development 

5.  Intangible  benefits  for  technical  staff. 

•  Rewards  and  recognition  for  excellence 

•  T echnical  support,  facilities 

•  Professional  development,  training 

In  addition  to  site  visits,  data  would  be  gathered  from  the  Services,  data  calls,  and  one-on-one 
interviews.  D  ata  quality  available  immediately  is  good. 

5.  State  of  Physical  Resources.  This  metric  is  intended  to  address  the  physical 
infrastructure  in  terms  of  physical  condition,  uniqueness,  and  potential  intangible  benefits  for 
the  S&Es. 

Recommended  evaluation  criteria  include: 

1 .  Support  of  unique  facilities  not  feasible  in  the  private  sector 

2.  Private  sector  facilities 

3.  0  pportunities  for  connectivity,  virtual  centers 

4.  Potential  facilities  duplication,  opportunities  for  consolidation,  leverage 

5 .  F acilities  as  intangible  benefits  fo  r  staff 

In  addition  to  site  visits,  data  would  be  gathered  from  the  Services  and  data  calls.  In  most 
cases,  data  quality  available  immediately  is  good. 

6.  State  of  Fiscal  Resources.  This  metric  is  intended  to  address  the  adequacy  of  the  current 
budget  to  support  a  healthy  D  0  D  S&T  and  R&D  program  into  the  future. 

Recommended  evaluation  criteria  include: 

1 .  S&T  funding  trends  -  total  and  as  percent  of  total  obligation  authority  (TO  A ) 

2.  RDT&E  funding  trends 

3.  TO  A  trends,  mix  of  non- RDT&E 

4.  Related  private  sector  funding  trends 

In  addition  to  site  visits,  data  would  be  gathered  from  the  Services,  data  calls,  IR&D  database, 
and  industry  association  data.  With  few  exceptions,  the  relevant  high  quality  data  is  available. 
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VI.  Timing  for  S&T  Invigoration 

In  the  early  days  of  the  Reagan  defense  buildup,  Secretary  of  Defense  Caspar  Weinberger 
wrote, 

We  must  recognize  that  our  current  advantages  come  principally  from  the  momentum 
in  basic  technologies  we  developed  during  the  1950s  and  1960s,  the  'golden  period'  of 
United  States  scientific  exploration.  We  face  the  danger  of  losing  our  edge  because  we 
have  not  adequately  replenished  the  reservoir  of  scientific  concepts  and  knowledge  to 
nourish  future  technologies  during  subsequent  years  of  fiscal  neglect  of  defense 
research  and  development.  Given  these  circumstances,  we  must  systematically 
replenish  that  scientific  reservoir,  using  the  unique  and  diverse  strengths  of  the  United 
States  scientific  community  ...  Given  the  relatively  long  lead  time  between 
fundamental  discovery  and  applying  such  knowledge  to  defense  systems,  the  true 
measure  of  our  success  ...  may  not  be  apparent  for  several  decades.  When  the 
'moment  of  truth'  arrives,  we  cannot  afford  to  be  found  wanting.  Thus  we  must 
revitalize  the  productive  partnership  between  the  university  community,  industry,  and 
the  D  0  D  in-house  laboratories.41 

Although  the  events  of  September  11,  2001  have  made  permanent  changes  to  the  American 
attitudes  toward  defense,  a  number  of  other  independent  factors  have  aligned  to  create  the 
right  timing  and  an  opportunity  to  consider  DOD  S&T  invigoration.  First  and  most 
important  is  the  immediacy  of  loss  of  irreplaceable  senior  staff.  Well-documented  from 
multiple  sources  is  the  fact  that  a  highly  talented  generation  hired  in  the  Viet  Nam  era  will 
retire  from  the  system  within  five  years,  without  trained  relief  to  preserve  the  corporate 
memory.  Today's  situation  was  created  through  1960s  rapid  expansion,  1970s-80s  hiring 
freezes,  1990s  downsizing  via  buyouts  and  other  retirement  incentives,  and  an  R&D  shift  to 
industry.  As  is  true  in  most  fields  of  endeavor,  scientists  quickly  become  aware  that  the  "art 
of  science  and  technology"  is  much  more  a  function  of  personal  hands-on  experience  and 
tutelage  of  veterans  than  of  formal  education.  Time  is  short  to  compensate  for  today's 
generation  gap  and  to  prevent  permanent  loss  of  capability  in  D  O  D -unique  technology  areas 
(energetic  materials,  hydrodynamics,  etc.).  In  a  March  14,  2001,  white  paper  on  the  Naval 
warfare  centers,  Dr.  Charles  Schoman  states  "The  Navy  is  in  danger  of  losing  a  capability 
which  it  has  taken  years  and  billions  of  dollars  and  millions  of  man-hours  to  develop  -  this  is 
its  in-house  technical  capability.  This  selected  capability  was  developed  because  it  was 
determined  the  Navy  needed  it.  O  nee  it  is  lost  it  probably  cannot  be  rebuilt.  The  danger  is 
great,  but  can  be  partially  and  quickly  corrected  by  internal  Navy  decision  and  actions. 
However,  time  is  running  out.  Both  people  and  facilities  don't  last  forever."42 

A  positive  factor  is  the  opportunity  afforded  by  the  D  O  D  reporting  requirement  contained  in 
Section  913  of  the  Conference  Report  accompanying  the  N ational  D  efense  Authorization  Act 
(NDAA)  for  Fiscal  Year  2000,  which  states:  "...the  Secretary  of  Defense  shall  develop  an 
appropriate  performance  review  process  for  rating  the  quality  and  relevance  of  work 
performed  by  the  Department  of  Defense  laboratories.  The  process  shall  include  customer 
evaluation  and  peer  review  by  Department  of  Defense  personnel  and  appropriate  experts 
from  outside  the  D  epartment  of  D  efense.  The  process  shall  provide  for  rating  all  laboratories 
of  the  Army,  Navy,  and  Air  Force  on  a  consistent  basis."43  The  Congressional  requirement 
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provides  both  an  excellent  opportunity  and  a  forcing  function  (1)  to  determine  the  state  of 
S&T  capability  within  the  DOD  S&T  enterprise,  and  (2)  to  develop  a  long-range  plan 
(including  both  public  and  private  sectors)  to  ensure  the  maintenance  of  essential  capabilities. 

An  additional  factor  is  the  change  in  administration,  which  normally  leads  to  an  evaluation  o  f 
the  infrastructure  and  policies  of  the  past  administration.  Three  D  efense  Science  Board  T ask 
Force  reports  published  in  the  year  2000  each  express  concern  for  the  loss  of  capability  within 
the  DOD  laboratory  system  and  propose  significant  remedial  actio  ns.44,45,46  These  reports 
echo  concerns  expressed  two  decades  ago  on  the  same  issues  and  restate  that  the  time  for 
action  is  now. 

Finally,  there  is  growing  recognition  in  the  military  and  across  the  public  and  private  sectors  of 
the  vital  importance  of  modernization,  scientific  discovery  and  innovation  to  the  U.S.  national 
defense  and  economic  interests.  Then  Secretary  of  the  Navy  Richard  Danzig  stated  in  the 
1999  Annual  DOD  Report  to  Congress  that  "...today's  readiness  is  being  preserved  at  the 
expense  of  tomorrow's  reguirements."47 

Vice  Admiral  D  ennis  McG  inn  discussed  the  importance  of  S&T  investments  to  encourage  the 
rapid,  low-cost  examination  of  emerging  "disruptive  technologies,"  in  a  March  9,  2001 D  efense 
D  aily  interview.  In  that  interview,  Admiral  McG  inn  stated,  "Innovation  and  new  ideas  are  in 
fact  what  is  reguired  in  this  dynamic  time." 48 

On  the  same  day,  Dr.  D.  Allan  Bromley  (Yale  physics  professor  and  S&T  advisor  to  the 
President  from  1989-1993)  expressed  a  similar  sentiment  from  a  different  perspective  in  a N  ew 
Y  ork  T imes  editorial.  D  r.  Bromley  stated, 

The  21st  century  economy  will  continue  to  depend  on  scientific  innovation. 
Economists  estimate  that  innovation  and  the  application  of  new  technology  have 
generated  at  least  half  of  the  phenomenal  growth  in  America's  gross  domestic  product 
since  World  War  II  ...  Technological  innovation  depends  upon  the  steady  flow  of 
discoveries  and  trained  workers  generated  by  federal  science  investments  in 
universities  and  national  laboratories.  These  discoveries  feed  directly  into  the 
industries  that  drive  the  economy.  It's  a  straightforward  relationship;  industry  is 
attentive  to  immediate  market  pressures,  and  the  federal  government  makes  the 
investments  that  ensure  long-term  competitiveness.49 
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VII.  An  Approach  for  Science  and  Technology  Invigoration 
Within  the  DOD  S&T  Enterprise 

As  discussed  in  the  introductory  section  of  this  report,  there  are  four  primary  issues  to  be 
addressed: 

1 .  D  eterminatio  n  o  f  the  fundamental  value  o  f  the  D  0  D  labo  rato  ries  a  nd  warfare  centers 
and  of  their  "highest  and  best"  uses,  as  they  exist  today. 

2.  E nsuring  the  technical  capabilities  of  the  D  0  D  S&T  enterprise  are  sufficiently  robust 
to  meet  the  D  epartment's  needs  of  today  and  in  the  foreseeable  future. 

3.  D  eterminatio n  of  what  is  reguired  of  the  D  0  D  S&T  enterprise  to  satisfy  the  needs  of 
the  DOD-after-next. 

4.  Creation  of  the  environment  to  draw  the  best  and  brightest  into  the  D  0  D  and  federal 
S&T  enterprise  of  the  future. 

To  address  fully  the  fundamental  value  of  the  DOD  laboratories  and  centers  and  their 
"highest  and  best"  uses  in  today's  environment  and  in  the  future  reguires  a  comprehensive 
top-down  strategy  for  RDT&E  that  exists  only  in  part.  An  excellent  Office  of  Net 
Assessment  Summer  Study  of  1992  conducted  for  the  Under  Secretary  of  Defense  (Policy) 
and  led  by  Andrew  W.  Marshall  and  James  G .  Roche  provides  useful  insights  for  evaluation 
criteria.  Four  core  competencies  were  identified  (based  on  the  Prahalad  and  Hamel  model50) 
that  characterized  the  D  epartment  of  D  efense,  including  the: 

1.  Ability  to  manage  across  the  broad  organization  of  the  DOD  that  consists  of  many 
diverse  and  sometimes  competing  sub-units  (command  and  control  and  logistics); 

2.  Ability  to  link  separate  streams  of  advanced  technologies; 

3.  Ability  to  manage  diverse  constituencies  and  needs  (training,  morale,  human  relations); 

4.  Capability  to  establish  a  professional  armed  force,  provide  leadership  for  it,  develop 
operational  art  and  implement  joint  and  combined  operations  ("self  awareness"  within 
the  forces  that  encourages  and  rewards  integrity,  innovation  and  professionalism). 51 

The  second  identified  competency,  the  ability  to  link  separate  streams  of  advanced 
technologies,  "is  manifest  in  systems  engineering  and  architecture,  in  communications,  in 
intelligence,  in  global  surveillance,  in  modeling  and  simulation,  and  in  high  performance 
technologies."  Following  the  Prahalad  and  Hamel  model,  the  physical  embodiment  of  one  or 
more  core  competencies  is  a  core  product,  which  for  DOD  includes  products  such  as 
avionics,  fire  control  systems,  positioning  systems,  acoustic  and  other  sensing,  propulsion 
technologies,  guidance  systems,  display  systems,  and  tactical  training.52  This  competency  is 
linked  especially  closely  with  effective  in-house  RDT&E  activities. 

A  logical  approach  in  assuring  the  relevance  of  the  Service  warfare  centers  for  today's 
environment  would  couple  an  objective  core  competency  assessment  with  reguirements  input 
from  the  Service  Chiefs,  Systems  Commands,  and  CINCs.  This  approach  could  address  also 
the  second  major  issue,  ensuring  sufficient  technical  capability  within  the  warfare  center 
system,  through  the  addition  of  a  management  and  technical  guality  assessment,  including 
customer  input  and  peer  review  from  within  and  outside  the  D  0  D . 
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The  third  issue,  i.e.  to  determine  the  requirements  of  the  enterprise  for  the  future,  while 
difficult,  is  addressed  also  in  the  Marshall  and  Roche  study.  This  study  introduced  the  term 
"military  after  next"  as  an  analogue  to  the  "factory  after  next"  concept  of  Professor  Joseph 
Bower  of  the  Harvard  Business  School.  The  report  states  "the  'next  factory'  will  provide  a 
marginal  increase  in  technology  to  produce  marginal  increases  in  operating  efficiency  and  cost- 
effectiveness  over  that  which  already  exists.  A  better  strategy  is  to  bypass  that  next  generation 
of  plant  and  to  build  'the  factory  after  next,'  the  one  that  represents  a  generational  step  in 
terms  of  technologies,  operational  effectiveness  and  operational  concepts.  Therefore,  the 
'factory  after  next'  will  lealize  increases  in  effectiveness  and  productivities  that  'the  next 
factory'  could  never  match  and  can  never  achieve."  The  report  then  recommends  that,  "What 
we  have  to  do  now  is  to  begin  our  planning  for  the  'military  after  next'  that  will  be  able  to 
cope  with  the  adversaries  of  any  class.  0  nee  we  have  spent  ten  years  doing  business  as  usual, 
making  marginal  improvements  in  the  capabilities  of  our  forces,  it  is  going  to  be  too  late  to  try 
and  reorient  our  thinking  and  catch  up  to  the  strategic  problem  that  we  may  face  in  2012." 
Three  new  competencies  are  suggested  for  the  future:  (1)  "the  ability  to  lead,  plan  and 
manage  the  process  of  using  a  wide  range  of  political,  economic  and  military  instruments  for 
shaping  the  future  security  environment;"  (2)  "the  in-depth  understanding  of  relevant  regions 
and  cultures;"  and  (3)  the  ability  to  better  utilize  the  global  economic  infrastructure  in  order  to 
fully  draw  upon  the  U.S.  and  international  infrastructures  for  commercial  components  in 
military  systems,  for  technology,  and  for  offshore  production  of  military  components  or 
systems  where  appropriate."53 

From  the  above  discussion,  one  may  conclude  that  an  especially  effective  DOD  technical 
infrastructure  of  the  future  would  be  flexible,  innovative,  willing  to  promote  bypassing  the 
next  generation  of  technology,  and  able  to  support  the  military  forces  in  effective  utilization  of 
the  systems  and  technologies  of  both  U.S.  and  international  military  and  commercial  activities. 
These  objectives  probably  can  be  accomplished  within  the  current  command  structures  of  the 
Services,  provided  that  a  leadership  for  a  collective  strategic  vision  is  provided  by  D  D  R&E . 
Such  a  structure  allows  the  best  of  both  evolutionary  and  revolutionary  worlds  to  coexist  and 
to  derive  strength  from  their  differing  perspectives. 

The  fourth  issue  to  be  addressed  is  creation  of  an  environment  to  draw  the  best  and  brightest 
into  DOD  and  federal  S&T  enterprise  of  the  future.  One  major  external  factor,  employment 
opportunities  for  S&Es  in  the  private  technology  sector,  may  be  changing  in  favor  of  the 
DOD  in  a  manner  very  similar  to  the  late  1960s  time  period  as  the  economy  cools.  If  so,  a 
small  window  of  opportunity  will  exist  again  to  bring  in  entry-level  S&Es  from  an  above- 
average  talent  pool. 

If  the  current  situation  leads  to  an  influx  of  young  talent,  it  is  especially  important  for  the 
national  technology  base  interests  that  the  new  crop  is  nurtured  and  provided  incentives  to 
grow  and  to  innovate.  Thus  creation  of  opportunities  to  encourage  innovation  is  vital.  The 
past  success  of  the  Navy's  6.2  IED  and  the  current  success  (and  defensibility)  of  the  NRL 
Base  Program  call  for  the  creation  of  6.2  base  programs  at  DOD  laboratories  and  warfare 
centers  to  support  and  encourage  S&T  revitalization  in  the  areas  of  necessary  DOD  core 
competencies.  The  investment  strategy  would  need  to  be  closely  monitored  to  verify  the 
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funding  was  accomplishing  its  desired  goal.  If  not,  alternatives  for  maintaining  the 
competency  elsewhere  in  the  public  and/  or  private  sectors  would  need  to  be  explored  quickly, 
to  minimize  loss  of  the  technical  corporate  memory. 


41 


Intentionally  Blank 


42 


VIII.  Summary  Recommendations 

1 .  Conduct  an  assessment  of  the  D  0  D  laboratories  and  warfare  centers  consistent  with  the 
Section  913  requirements  to  determine  the  state  of  S&T  within  the  Department  of 
Defense.  Focus  on  the  relevant  DOD  competencies  and  the  requirements  input  of  the 
Service  Chiefs,  Systems  Commands  and  CINCs.  Metrics  for  assessment  should  include 
the  following: 

•  Q  uality  and  military  impact  of  the  products  of  the  organization; 

•  Core  Competencies  of  the  organization; 

•  V alue-added  of  the  organization; 

•  Q  uality  of  the  technical  staff; 

•  State  of  Physical  Resources;  and 

•  State  of  Fiscal  Resources. 

2.  Develop  a  strategy  to  ensure  maintenance  of  the  DOD  S&T  corporate  memory  as  the 
current  generation  of  senior  scientists  and  engineers  retires.  Examine  the  potential  for  a 
window  of  opportunity  to  recruit  the  "best  and  brightest"  into  DOD  and  defense- related 
industry  should  commercial  technology  sector  demand  decrease. 

3.  D  evelop  a  strategic  plan  for  the  D  0  D  S&T  infrastructure  of  the  future  that  defines  the 
complementary  roles  of  the  DOD  laboratories  and  warfare  centers,  academia, 
FFRD  Cs /  UARCs  and  private  industry.  The  future  system  should  be  flexible,  innovative, 
willing  to  promote  bypassing  the  next  generation  of  technology  and  able  to  support  the 
military  forces  in  effective  utilization  of  the  systems  and  technologies  of  both  U.S.  and 
international  military  and  commercial  activities. 

4.  Establish  and  monitor  closely  a  6.2  Base  Program  within  the  DOD  laboratories  and 
warfare  centers.  Consider  outyear  program  growth  as  appropriate,  depending  on  the 
degree  of  success  of  the  initial  effort. 

5.  D  D  R&E  should  take  the  lead  to  carry  out  the  above  recommendations. 
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Appendix  A. 

Text  of  Section  913,  Conference  Report,  National  Defense 
Authorization  Act  for  the  Fiscal  Year  2000 

SEC.  913.  EFFICIENT  UTILIZATION  OF  DEFENSE  LABORATORIES. 

(a)  ANALYSIS  BY  INDEPENDENT  PANEL.  -  (1)  not  later  than  45  days  after  the  date  of  the 
enhancement  of  this  A  ct,  the  Secretary  of  Defense  shall  convene  a  panel  of  independent  experts  under  the 
auspices  of  the  D  efense  Science  Board  to  conduct  an  analysis  of  the  resources  and  capabilities  of  all  the 
laboratories  and  test  and  evaluation  facilities  of  the  D  epartment  ofD  efense,  including  those  of  the  military 
departments.  In  conducting  the  analysis,  the  panel  shall  identify  opportunities  to  achieve  efficiency  by 
consolidating  lead  agencies  or  executive  agents  in  cases  considered  appropriate.  The  panel  shall  report  its 
findings  to  the  Secretary  of  D  efense  and  to  Congress  not  later  than  A  ugust  1,2000. 

(2)  The  analysis  required  by  paragraph  (1)  shall,  at  a  minimum,  address  the  capabilities  of  the  laboratories 
and  test  and  evaluation  facilities  in  the  areas  of  air  vehicles,  armaments,  command,  control  and 
communications,  and  intelligence,  space,  directed  energy,  electronic  warfare,  medicine,  corporate  laboratories,  civil 
engineering,  geophysics,  and  the  environment. 

(b)  PERFORMANCE  REVIEW  PROCESS.  -  Not  later  than  180  days  after  the  date  of  the 
enactment  of  this  A  ct,  the  Secretary  ofD  efense  shall  develop  an  appropriate  performance  review  process  for 
rating  the  quality  and  relevance  of  work  performed  by  the  D  epartment  ofD  efense  laboratories.  This  process 
shall  include  customer  evaluation  and  peer  review  byD  epartment  ofD  efense  personnel  and  appropriate  experts 
from  outside  the  D  epartment  ofD  efense.  T  he  process  shall  provide  for  rating  of  all  the  laboratories  of  the 
A  rmy,  N  avy,  and  A  ir  Force  on  a  consistent  basis. 
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